Papers with kernel principal component analysis
A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny (2025.acl-srw)
Copied to clipboard
| Challenge: | Recent studies suggest that self-attention implements kernel principal component analysis (KPCA) Across 10 transformer architectures, we conclude that the KPCA interpretation of self- attention lacks empirical support. |
| Approach: | They revisit claims that self-attention implements kernel principal component analysis . they argue that self attention projects queries onto principal component axes of key matrix K . |
| Outcome: | The proposed kernel principal component analysis does not match the proposed kernel . the proposed method is not able to detect the eigenvalues of the gram matrix . |
Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for gender bias mitigation for word embeddings are based on pre-trained word embeds . however, the assumption that the bias subspace is linear is untested . |
| Approach: | They propose a method to isolate gender bias in word embeddings using pre-trained word embeds. |
| Outcome: | The proposed method eliminates gender bias in word embeddings but assumes bias subspace is linear . the proposed method has some drawbacks, but it is a good one for a non-linear analysis. |